Why the Information Explosion Can Be Bad for Data Mining, and How Data Fusion Provides a Way Out

One may claim that the exponential growth in the amount of data provides great opportunities for data mining. Reality can be different though. In many real world applications, the number of sources over which this information is fragmented grows at an even faster rate, resulting in barriers to widespread application of data mining and missed business opportunities. Let us illustrate this paradox with a motivating example from database marketing. In marketing, direct forms of communication are becoming increasingly popular. Instead of broadcasting a single message to all customers through traditional mass media such as television and print, the most promising potential customers receive personalized offers through the most appropriate channels. So it becomes more important to gather information about media consumption, attitudes, product propensity etc. at an individual level. Basic, company specific customer information resides in customer databases, but market survey data depicting a richer view of the customer are only available for a small sample or a disjoint set of reference customers. Collecting all this information for the whole customer database in a single source survey would certainly be valuable but is usually a very expensive proposition. The common alternative within business to